
<!DOCTYPE html>
<!--[if IEMobile 7 ]><html class="no-js iem7"><![endif]-->
<!--[if lt IE 9]><html class="no-js lte-ie8"><![endif]-->
<!--[if (gt IE 8)|(gt IEMobile 7)|!(IEMobile)|!(IE)]><!--><html class="no-js" lang="en"><!--<![endif]-->

<head>


<meta charset="utf-8">
<meta http-equiv="cleartype" content="on">

<title>How to Debug Hadoop on Local - Sam小龙</title>
<meta name="author" content="xiaolong.yuanxl">




<meta name="description" content="伪分布式debug hadoop源码">

<meta name="keywords" content="hadoop ">


<!-- http://t.co/dKP3o1e -->
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
<meta name="viewport" content="width=device-width, initial-scale=1">

<!-- Twitter Cards -->


<!-- Open Graph -->
<meta property="og:local" content="en_US">
<meta property="og:type" content="article">
<meta property="og:url" content="http://yuanxiaolong.github.io/blog/2014/07/21/how-to-debug-hadoop-on-local">
<meta property="og:title" content="how to debug hadoop on local">
<meta property="og:description" content="伪分布式debug hadoop源码">

  <meta property="og:image" content="">

<meta property="og:site_name" content="Sam小龙">

<link rel="canonical" href="http://yuanxiaolong.github.io/blog/2014/07/21/how-to-debug-hadoop-on-local">
<link href="/favicon.png" rel="icon">
<link href="/stylesheets/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
<!-- <link href="//netdna.bootstrapcdn.com/font-awesome/4.1.0/css/font-awesome.min.css" rel="stylesheet"> -->
<link href="/stylesheets/font-awesome.min.css" rel="stylesheet">
<link href="/atom.xml" rel="alternate" title="Sam小龙" type="application/atom+xml">

<!-- the flow button on left -->
<script src="/javascripts/vendor/modernizr-2.6.2.custom.min.js"></script>


  <style type="text/css">body {background-image:url(/images/bg.jpg);background-size:cover;}</style>

<!-- 
  

  
    
  

  <style type="text/css">body {background-image:url(
      /images/
    
      bg.jpg
    
  
    );}</style>
 -->

<!--Fonts from Google"s Web font directory at http://google.com/webfonts
<link href="//fonts.googleapis.com/css?family=PT+Serif:regular,italic,bold,bolditalic" rel="stylesheet" type="text/css">
<link href="//fonts.googleapis.com/css?family=PT+Sans:regular,italic,bold,bolditalic" rel="stylesheet" type="text/css">
-->


</head>

<body id="post" >

<!--[if lt IE 9]><div class="upgrade"><strong><a href="http://whatbrowser.org/">Your browser is quite old!</strong> Why not upgrade to a different browser to better enjoy this site?</a></div><![endif]-->
<nav id="dl-menu" class="dl-menuwrapper" role="navigation">
	<button class="dl-trigger">Open Menu</button>
	<ul class="dl-menu">
		<li><a href="/">主页</a></li>

		<li>
			<a href="#">内容</a>
			<ul class="dl-submenu">
				<li><a href="/posts/">文章列表</a></li>
				<li><a href="/categories/">分类列表</a></li>
			</ul>
		</li>

		<li>
			<a href="#">关于</a>
			<ul class="dl-submenu">
				<li>
					<img src="/images/avatar.png" alt="xiaolong.yuanxl photo" class="author-photo">
					<h4>xiaolong.yuanxl</h4>
					<p></p>
				</li>
				<li><a href="/about">关于我</a></li>
				<li>
					<a href="mailto:232351936@qq.com"><i class="fa fa-envelope"></i> Email</a>
				</li>
				
				
				<li>
					<a href="http://github.com/yuanxiaolong"><i class="fa fa-github"></i> GitHub</a>
				</li>
			</ul><!-- /.dl-submenu -->
		</li>

		
	</ul><!-- /.dl-menu -->
</nav><!-- /.dl-menuwrapper -->




<div id="main" role="main" >
  <article class="hentry">
    <header class="header-title">
      <div class="header-title-wrap">
        
          <h1 class="entry-title"><a href="/blog/2014/07/21/how-to-debug-hadoop-on-local/" rel="bookmark" title="how to debug hadoop on local">how to debug hadoop on local</a></h1>
        
        <h2>July 21, 2014</h2>
      </div><!-- /.header-title-wrap -->
    </header>
    <div class="entry-content">
      <p>介绍一下如何在伪分布式hadoop上，配置hadoop源码debug环境，以便学习</p>

<!--more-->

<p>本文分2部分来介绍</p>

<ol>
  <li>配置源码显示，以便跟踪</li>
  <li>创建远程debug hadoop的jvm</li>
</ol>

<hr />

<h2 id="section">环境准备</h2>

<ol>
  <li>本地部署伪分布式hadoop（可以参考 <a href="http://blog.yuanxiaolong.cn/blog/2014/07/12/install-hadoop-on-local/" target="_blank"><font color="#6868b4">这里</font></a>）</li>
  <li>下载hadoop源码，2种方式。
    <ul>
      <li>svn <a href="http://svn.apache.org/repos/asf/hadoop/common/tags/release-1.2.1/" target="_blank"><font color="#0c95a1"> http://svn.apache.org/repos/asf/hadoop/common/tags/release-1.2.1/ </font></a></li>
      <li>git <a href="https://github.com/apache/hadoop-common/tree/release-1.2.1" target="_blank"><font color="#0c95a1"> github 地址 </font></a></li>
    </ul>
  </li>
  <li>eclipse 配置已好，标准的就可以，可以在 <a href="http://www.eclipse.org/downloads/" target="_blank"><font color="#6868b4">这里</font></a> 下载</li>
</ol>

<hr />

<h2 id="eclipse-hadoop">配置eclipse hadoop源码工程</h2>

<font color="#7c837f" size="3"> 为了方便查看debug时所在的行号、变量、类什么的，我们需要先配置hadoop源码工程到eclipse里。</font>

<p>1.解压源码包，将源码包里的这3个目录下的文件copy出来，放到一个新的org/apache/hadoop文件夹内，准备导入到eclipse里。</p>

<ul>
  <li>src/core/org/apache/hadoop</li>
  <li>src/hdfs/org/apache/hadoop</li>
  <li>src/mapred/org/apache/hadoop</li>
</ul>

<p>2.eclipse里新建一个java工程，导入刚才合并后的 org/apache/hadoop 内的文件 <font color="#7c837f">（此时刷新工程，会有很多红叉）</font></p>

<p>3.将hadoop源码里的lib文件夹下的所有jar包 copy出来 <font color="#7c837f"> (同时网上找一个 </font><a href="http://pan.baidu.com/s/1bnlI8EV" target="_blank"><font color="#6868b4">ant.jar</font></a> <font color="#7c837f">添加进去)</font>，放到刚才我们新建的工程里，并配置eclipse build path 依赖这些jar</p>

<p>4.此时，还有1个报错，是因为eclipse默认禁止使用sun的jar包，需要在buildpath中添加一个规则，右击工程 <font color="#d66d58">属性-&gt;buildpath</font>，如下：
<img src="/images/hadoop/eclipse-hadoop-src-sun-jar-error.png" alt="sun" /></p>

<hr />

<h2 id="debug">开始debug</h2>

<p>1.修改 hadoop-env.sh，在8001端口上，开启对namenode 的jvm调试。<font color="#7c837f">（如果学过j2ee，跟tomcat远程调试一样的道理）</font></p>

<div class="bogus-wrapper"><notextile><figure class="code"><figcaption><span>hadoop-env.sh</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
</pre></td><td class="code"><pre><code class="bash"><span class="line"><span class="c"># Command specific options appended to HADOOP_OPTS when specified</span>
</span><span class="line"><span class="c">#export HADOOP_NAMENODE_OPTS=&quot;-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS&quot;</span>
</span><span class="line"><span class="nb">export </span><span class="nv">HADOOP_NAMENODE_OPTS</span><span class="o">=</span><span class="s2">&quot;-Xdebug -Xrunjdwp:transport=dt_socket,address=8001,server=y,suspend=y&quot;</span>
</span><span class="line"><span class="nb">export </span><span class="nv">HADOOP_SECONDARYNAMENODE_OPTS</span><span class="o">=</span><span class="s2">&quot;-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS&quot;</span>
</span><span class="line"><span class="nb">export </span><span class="nv">HADOOP_DATANODE_OPTS</span><span class="o">=</span><span class="s2">&quot;-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS&quot;</span>
</span><span class="line"><span class="nb">export </span><span class="nv">HADOOP_BALANCER_OPTS</span><span class="o">=</span><span class="s2">&quot;-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS&quot;</span>
</span><span class="line"><span class="nb">export </span><span class="nv">HADOOP_JOBTRACKER_OPTS</span><span class="o">=</span><span class="s2">&quot;-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS&quot;</span>
</span></code></pre></td></tr></table></div></figure></notextile></div>

<p>2.在我们eclipse hadoop源码工程中，打开 <font color="#a6752f"> NameNode.java </font> ，在280行打上一个断点。</p>

<p>3.在eclipse里新建远程调试<font color="#7c837f">（先别点debug按钮，因为此时hadoop还没运行）</font>
<img src="/images/hadoop/hadoop-eclipse-debugconf.png" alt="eclipse-debug-conf" /></p>

<p>4.运行hadoop，可以看到在8001开启了端口用于debug
<img src="/images/hadoop/hadoop-start-debug-port.png" alt="eclipse-debug-conf" /></p>

<p>5.此时，再点击hadoop工程的debug远程调试按钮，可以看到，断点已经生效
<img src="/images/hadoop/hadoop-breakpoint.png" alt="eclipse-debug-conf" /></p>

<hr />

<h2 id="debug-shell-command">debug shell command</h2>

<p>1.修改 bin/hadoop 脚本。</p>

<div class="bogus-wrapper"><notextile><figure class="code"><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
</pre></td><td class="code"><pre><code class="bash"><span class="line"><span class="k">elif</span> <span class="o">[</span> <span class="s2">&quot;$COMMAND&quot;</span> <span class="o">=</span> <span class="s2">&quot;fs&quot;</span> <span class="o">]</span> <span class="p">;</span> <span class="k">then</span>
</span><span class="line"><span class="nv">CLASS</span><span class="o">=</span><span class="s1">&#39;org.apache.hadoop.fs.FsShell&#39;</span>
</span><span class="line"><span class="nv">HADOOP_OPTS</span><span class="o">=</span><span class="s2">&quot;$HADOOP_OPTS $HADOOP_CLIENT_OPTS&quot;</span>
</span></code></pre></td></tr></table></div></figure></notextile></div>

<p>在下面多加一句，然后重新启动Hadoop集群，这样在命令行的所有操作都可以在FsShell里断点了。</p>

<div class="bogus-wrapper"><notextile><figure class="code"><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
</pre></td><td class="code"><pre><code class="bash"><span class="line"><span class="nv">HADOOP_OPTS</span><span class="o">=</span><span class="s2">&quot;$HADOOP_OPTS $HADOOP_CLIENT_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8008&quot;</span>
</span></code></pre></td></tr></table></div></figure></notextile></div>


      <!-- add copyright for each post -->
      <div style="margin-top:20px;">
        <font color="gray" size="3">
          原文地址，转载请注明出处:<br/>
          <a href="http://blog.yuanxiaolong.cn/blog/2014/07/21/how-to-debug-hadoop-on-local/">http://blog.yuanxiaolong.cn/blog/2014/07/21/how-to-debug-hadoop-on-local/</a>
        </font>
      </div>
      <footer class="entry-meta">
        <span class="entry-tags"><a href="/categories/#hadoop" title="Pages tagged hadoop" class="tag">hadoop</a></span>
        <span><a href="/blog/2014/07/21/how-to-debug-hadoop-on-local/" rel="bookmark" title="how to debug hadoop on local">how to debug hadoop on local</a> was published on <span class="entry-date date published updated"><time datetime="2014-07-21T20:59:42+08:00">July 21, 2014</time></span></span>
        
        <span class="author vcard"><span class="fn"><a href="" title="About xiaolong.yuanxl">xiaolong.yuanxl</a></span></span>
        <div class="social-share">
          <div class="socialcount socialcount-small inline-list">
            <!-- <li class="facebook"><a href="https://www.facebook.com/sharer/sharer.php?u=/blog/2014/07/21/how-to-debug-hadoop-on-local/" title="Share on Facebook"><span class="count"><i class="fa fa-facebook-square"></i> Like</span></a></li>
            <li class="twitter"><a href="https://twitter.com/intent/tweet?text=/blog/2014/07/21/how-to-debug-hadoop-on-local/" title="Share on Twitter"><span class="count"><i class="fa fa-twitter-square"></i> Tweet</span></a></li>
            <li class="googleplus"><a href="https://plus.google.com/share?url=/blog/2014/07/21/how-to-debug-hadoop-on-local/" title="Share on Google Plus"><span class="count"><i class="fa fa-google-plus-square"></i> +1</span></a></li> -->
            <!-- 多说评论框 start -->
          	<div class="ds-thread" data-thread-key="/blog/2014/07/21/how-to-debug-hadoop-on-local/" data-title="how to debug hadoop on local" data-url="http://blog.yuanxiaolong.cn/blog/2014/07/21/how-to-debug-hadoop-on-local/"></div>
            <!-- 多说评论框 end -->
            <!-- 多说公共JS代码 start (一个网页只需插入一次) -->
            <script type="text/javascript">
            var duoshuoQuery = {short_name:"yuanxiaolong"};
            	(function() {
            		var ds = document.createElement('script');
            		ds.type = 'text/javascript';ds.async = true;
            		ds.src = (document.location.protocol == 'https:' ? 'https:' : 'http:') + '//static.duoshuo.com/embed.js';
            		ds.charset = 'UTF-8';
            		(document.getElementsByTagName('head')[0]
            		 || document.getElementsByTagName('body')[0]).appendChild(ds);
            	})();
            	</script>
            <!-- 多说公共JS代码 end -->
        </div>
        </div><!-- /.social-share -->
      </footer>

    </div><!-- /.entry-content -->
    
      <div class="read-more">
        
          <div class="read-more-header">
            <a href="/blog/2014/09/07/install-nfs-and-share-file/" class="btn">Read More</a>
          </div><!-- /.read-more-header -->
          <div class="read-more-content">
            <h3><a href="/blog/2014/09/07/install-nfs-and-share-file/" title="install nfs and shshare file">install nfs and shshare file</a></h3>
            <p>安装nfs服务器，用于共享配置 <a href="/blog/2014/09/07/install-nfs-and-share-file/"> Continue reading</a></p>
          </div><!-- /.read-more-content -->
        
        <div class="read-more-list">
          
            <div class="list-item">
              <h4><a href="/blog/2014/09/07/install-dns-server-bind/" title="install dns server bind">install dns server bind</a></h4>
              <span>Published on September 07, 2014</span>
            </div><!-- /.list-item -->
          
            <div class="list-item">
              <h4><a href="/blog/2014/08/31/analysis-hadoop-load-configuration/" title="analysis hadoop load configuration">analysis hadoop load configuration</a></h4>
              <span>Published on August 31, 2014</span>
            </div><!-- /.list-item -->
          
        </div><!-- /.read-more-list -->
      </div><!-- /.read-more -->
    
    
  </article>
</div><!-- /#main -->

<div class="footer-wrapper">
  <footer role="contentinfo">
    <span>&copy; 2014 xiaolong.yuanxl. Powered by <a href="http://octopress.org">Octopress</a> using the <a href="https://github.com/Z1MM32M4N/hpstr-theme/">HPSTR Theme for Octopress</a>.</span>

  </footer>
</div><!-- /.footer-wrapper -->



<script src="http://cdn.staticfile.org/jquery/1.9.1/jquery.min.js"></script>
<script>window.jQuery || document.write('<script src="/javascripts/vendor/jquery-1.9.1.min.js"><\/script>')</script>
<!--
<script src="/javascripts/octopress.js" type="text/javascript"></script>
-->
<script src="/javascripts/scripts.min.js"></script>
<!-- 
 -->
<script type="text/javascript" src="http://tajs.qq.com/stats?sId=35560703" charset="UTF-8"></script>






</body>
</html>
